Compression of surface light fields

ABSTRACT

A system and method are provided for compressing a graphical representation that describes the appearance of an object from a plurality of viewing directions. Compressing includes accessing the graphical representation, removing redundant descriptive information from the graphical representation and representing the remaining information for efficient rendering, such as rendering with hardware-assisted computation.

BACKGROUND OF THE INVENTION

[0001] 1. Field of the Invention

[0002] The invention relates generally to compressing graphical data forthree-dimensional objects. More particularly, the invention relates to asystem and method for efficiently compressing graphical data thatdescribes the appearance of a three-dimensional object from multipleviewing directions.

[0003] 2. Background Information

[0004] Accurate portrayal of the physical world has been an importantgoal of computer graphics since its inception. This includes modelingthe geometry and surface attributes of the object when the object isilluminated by a light source and viewed in a virtual environment from acertain vantage point. Typically, the geometry is modeled as a polygonalmesh that models the geometric extents of the object. The surfaceattributes, which can be modeled in a number of ways, are then appliedto the geometry during rendering to graphically display the object.

[0005] The essential part of the quest for realistic graphics is propermodeling of the visual attributes of the surfaces of objects. Forexample, proper modeling of surfaces that have luster, reflections, suchas chrome reflections, semi-transparent regions, complex textures likefur, and other qualities, is needed to create realistic visual models ofmany real-world objects. Theoretical and empirical approaches arepossible.

[0006] Historically, parametric reflectance models, such as Phongshading, have been favored since they are simple and compact. Howeverthese techniques can be inaccurate, are difficult to implement ontraditional graphics hardware, and are challenging to develop. Analternative is to use image-based techniques. Such image-basedtechniques offer a simple method of acquiring the visual data and anaccurate method for portraying the physical world. They representdiscrete visual data, acquired from the physical world using a varietyof imaging devices, directly in the sample-based format withoutresorting to the parametric models.

[0007] However the amount of data associated with such models isenormous. This significantly limits their use, since storage, datatransmission, and rendering suffer as a result of the large size ofthese models. Many images are needed to display the object fromdifferent vantage points without creating artifacts. Some approachesinterpolate between two related images to reduce the amount of datarequired, however this approach does not significantly reduce the amountof data needed, and many images are still required to displayvisually-realistic objects.

BRIEF DESCRIPTION OF THE SEVERAL VIEWS OF THE DRAWINGS

[0008] The novel features believed characteristic of the invention areset forth in the appended claims. The present invention is illustratedby way of example, and not by way of limitation, in the figures of theaccompanying drawings and in which like reference numerals refer tosimilar elements. The invention itself, however, as well as a preferredmode of use, will best be understood by reference to the followingdetailed description of an illustrative embodiment when read inconjunction with the accompanying drawings:

[0009]FIG. 1 shows a method for displaying an approximate graphicalrepresentation of an object, according to one embodiment of the presentinvention.

[0010]FIG. 2 shows a method for creating approximate graphicalrepresentations and using the approximate graphical representations todisplay objects, according to one embodiment of the present invention.

[0011]FIG. 3 shows a unitary viewing vector expressed in atriangle-centered reference frame, according to one embodiment of thepresent invention.

[0012]FIG. 4 shows summation of an umbrella function for a giventriangle, according to one embodiment of the present invention.

[0013]FIG. 5 shows a unitary viewing vector expressed in avertex-centered reference frame, according to one embodiment of thepresent invention.

[0014]FIG. 6 shows converting viewing directions into texturecoordinates, according to one embodiment of the present invention.

[0015]FIG. 7 shows a view texture map and surface texture map fortriangle-centered decomposition, according to one embodiment of thepresent invention.

[0016]FIG. 8 shows view texture maps and surface texture maps forvertex-centered decomposition, according to one embodiment of thepresent invention.

[0017]FIG. 9 shows an exemplary system for implementing the inventionover the Internet, according to one embodiment of the present invention.

[0018]FIG. 10 is a block diagram of a computer system upon which oneembodiment of the present invention may be implemented.

DETAILED DESCRIPTION OF THE INVENTION

[0019] In the following description, for the purpose of explanation,numerous specific details are set forth in order to provide a thoroughunderstanding of the present invention. It will be apparent, however, toone skilled in the art that the present invention may be practicedwithout some of these specific details. In other instances, well-knownstructures and devices are shown in block diagram form.

[0020] The present invention includes various steps, which will bedescribed below. The steps of the present invention may be performed byhardware components or may be embodied in machine-executableinstructions, which may be used to cause a general-purpose orspecial-purpose processor or logic circuits programmed with theinstructions to perform the steps. Alternatively, the steps may beperformed by a combination of hardware and software.

[0021] Displaying An Approximate Representation of an Object

[0022]FIG. 1 shows a method for displaying an object using anapproximate graphical representation of the object, according to oneembodiment of the present invention. Block 110 shows accessing agraphical representation of the appearance of an object, such as a realobject or a computer-generated object. Accessing may include loading therepresentation from a memory location or computer readable medium.Typically, the representation will include a geometry representation,which describes the geometry of the object, and a reflectancerepresentation, which describes the reflective properties of the object.

[0023] The geometry representation will typically be polygonalrepresentation, since many graphics pipelines are adept at renderingsuch representations. For example, the geometry representation may be atriangular mesh that defines the geometric extents of the object. Thegeometry representation may be created using computer graphics software,such as a wire-frame model created for a fictional animated character,or may be empirically determined for a real object by using cameras,scanners, range sensing equipment, and mesh editing software. Othergeometric representations are possible.

[0024] The reflectance representation, which may also be called asurface light field, describes the visual appearance of the object, suchas the radiance of the object or the light rays reflected by the object.The reflectance representation may describe the appearance when theobject is viewed from multiple viewing directions. Frequently, theappearance representation will be a multidimensional appearancerepresentation that includes one or more parameters sufficient todescribe a location on the surface of the object, and one or moreparameters sufficient to describe a viewing direction that the object isviewed from. For example, the appearance representation may be afour-dimensional representation, either a function or a data set, withtwo parameters to describe a position on the surface of the object andtwo parameters to describe a viewing direction. The appearance may varyfrom point to point and viewing direction to viewing direction, sincethis is typically the case for real objects.

[0025] Additional parameters may be included, such as to specify alocation on the interior of the object, to specify motion of the objectin space, and for other reasons that will be apparent to those skilledin the art. Since many real-world objects are viewed under constantlighting, such as an object illuminated by the sun or by fixed lightingin a room, the present disclosure will emphasize fixed lighting.However, the invention is equally applicable to variable lightingconditions, such as when lighting varies with time, location on theobject, or viewing direction. Each of these additional factors adds adimension to the graphical representation, so that representations withsix or more dimensions may be common. Typically, multidimensionalrepresentations of six or higher dimensions become complicated to storeand manipulate, requiring significant computing memory andcommunications capability.

[0026] The reflectance representation may be based on analyticalapproaches, such as parametric reflectance models, or empiricalapproaches, such as image-based models. In the case of the analyticalapproaches, the graphical representation will typically be a function,or a discrete data structure generated based on the function. Forexample, the reflectance representation may be a parametric reflectancemodel. In the case of the empirical approaches, the graphicalrepresentation will typically be a set of images taken of the objectfrom different viewing directions, or a data structure based on the setof images. The reflectance representation will typically be separatedfrom the geometry representation, although they may be combined in otherimplementations.

[0027] The invention will primarily be described in the context of ageometry representation that is a triangular mesh and a reflectancerepresentation that is determined empirically by acquiring data for areal object from different viewing directions under constant lightingconditions. Those skilled in the art will recognize that the inventionis equally applicable to other geometry and reflectance representations.According to one embodiment, data acquisition is performed on a realobject by taking pictures of the object from different viewingdirections with a hand-held digital camera. The object is positioned ona platform with indicia or markings to assist in identifying the cameralocation. The camera location for each picture is determined based onthe indicia. Between 150-300 images are collected of the object,although this number depends on the complexity of the object, such asthe symmetry of the object and the level of detail, and the accuracy andquality of the representation. Software is then used to register thetriangular mesh to each of the digital images. This registration can beperformed with very low error, such as one or a few pixels.Preprocessing may also be performed on these images, such as to filternoise, put the data in a set that facilitates subsequent analysis, andto standardize the data.

[0028] Block 120 shows creating an approximate graphical representationbased on the original graphical representation. Typically, thereflectance representation will be approximated, since it has moreinformation than the geometry representation, although the geometryrepresentation may also be approximated. The approximate representationmay be created by compressing the original representation so that therepresentation may be stored in less total memory and transferred fasteror with less bandwidth. Compressing may be done by removing redundancyfrom the set of digital images corresponding to different viewingdirections. For example, important information from the set of digitalimages may be selectively represented in the approximate reflectancerepresentation and redundant information may be selectively removed. Theimportant information may be tied to heterogeneity in the object, sincesymmetry, similar surface attributes, and other features may addredundancy that can be selectively removed. If the information betweendifferent views is similar one can chose to retain the information onlyonce. Specular and diffuse information may be separated so that only thechanging information may be selectively represented. The compression maybe done without spectral analysis of the appearance representation, suchas that which is done in DCT approaches, and may be done withoutcreating a codebook, such as in vector quantization.

[0029] In one embodiment, creating an approximate graphicalrepresentation may include creating a representation with at least oneexpression having fewer dimensions than an original multi-dimensionalgraphical representation by decomposing the original multi-dimensionalrepresentation. In one embodiment, all of the expressions in theapproximate graphical representation have fewer dimensions than theoriginal multi-dimensional graphical representation, although when theyare combined, the combination may have the same number of dimensions asthe original graphical representation. Decomposing may either select asubset of the most relevant information from the original representationor re-represent the relevant information.

[0030] Decomposing is to be interpreted broadly. Decomposing may includeconsolidating an n-dimensional cloud of disparate values, whether theyare truly disparate or disparate due to noise or error, into alower-dimensional approximation. For example, decomposing may includeregression and other forms of statistical consolidation. Decomposingthis may include linearly regressing a two-dimensional cloud ofdisparate values as a linear expression of a predictor. That is, atwo-dimensional set of data may be approximated with a one-dimensionalexpression, such as a regression line.

[0031] Decomposing may also include projecting a multi-dimensionalrepresentation into a lower-dimensional subspace. According to oneembodiment, singular value decomposition is used to decompose a matrixby mathematically approximating each column of the matrix by itsorthogonal projection onto the linear subspace spanned by K vectors,where K is an integer greater than or equal to one and that may beincreased to improve the approximation of the original representation.The number of vectors K may be varied to provide varying levels ofapproximation. The original reflectance representation is firstexpressed as a matrix, such as by rearranging discrete data, then thematrix is decomposed, using singular value decomposition, into atruncatable sum of products of vectors. These vectors may be expressedas two-dimensional matrices, such as by rearrangement, to facilitaterendering. This will be explained more fully below.

[0032] Creating may include creating an approximate reflectancerepresentation that is efficiently or quickly manipulated usingmultiprocessing or hardware operations. For example, the approximategraphical representation may be in a format supported by matrixoperations, multitexturing operations, and other operations that allowmore data and more complicated operations to be handled simultaneouslyor in fewer computer operations. For example, the approximaterepresentation may include a first matrix portion to be multiplied by asecond matrix portion, which can be quickly and efficiently performedusing multitexturing hardware that allows multiple textures to beconcurrently applied to a primitive in a single rendering pass. Ratherthan using hardware, the same format can be concurrently combined ormultiplied using multiprocessing, in which multiple processors ormultiple computers on a network may simultaneously process or combinethe portions. According to one embodiment, the operations to removeredundancy produce data structures that may be combined directly,without decompression, to create an appearance for the object. Thisdiffers from other approaches such as run-length approaches, JPEG, andothers where the data structures cannot be combined directly until eachare first decompressed.

[0033] Creating may include creating a viewing-dependent portion, whichstores information for different viewing directions, and aposition-dependent portion, which stores information for positions onthe object (e.g., a triangle). The view-dependent portion may includeview-dependent subportions that each correspond to a particular viewingdirection, such that one of the view-dependent subportions isconcurrently combined with the position-dependent portion to visualizethe object from the particular viewing direction. The portions may besubstantially different from images of the object take from differentviewing directions and may be combined without interpolation to displaythe object with one appearance for one viewing direction and a differentappearance for another viewing direction. For example, the portions maybe matrices, which correspond to substantial manipulation and processingof images of the object from different viewing directions, that may bemultiplied without interpolation using multitexturing hardware.

[0034] Block 130 shows rendering the object using the approximatedreflectance representation. Rendering may include accessing theapproximate reflectance representation, such as from a memory location,and concurrently processing multiple portions of the approximatereflectance representation to create an appearance for the object or aportion of the object. The portions may have the characteristicsdiscussed above, such as the fact that they may be combined with orwithout compression, some may be position-dependent and othersviewing-dependent, the viewing-dependent portion may have subportionscorresponding to particular viewing directions, and other features. Theobject may then be rendered and displayed by applying the appearances tothe object and displaying the object on a display device.

[0035] Exemplary Method For Approximating the Object Representation

[0036]FIG. 2 shows an exemplary method for creating approximategraphical representations and using the approximate graphicalrepresentations to display objects, according to one embodiment of thepresent invention. According to one embodiment a group of blocks 200 maybe performed ahead of time and a second group of blocks 255 may beperformed at a different time or in real time. The blocks 200 may alsobe performed on a first computer system and the blocks 255 performed ona second computer system, such as if the approximations are created onthe first computer system and distributed on computer-readable medium ortransmitted via a transmission medium to the second computer system.

[0037] At block 210 image-based data that describes the appearance of anobject from multiple viewing directions is accessed, such as from amemory. For example, the image-based data may be digital images of theobject acquired from different known viewing directions as well asacquired geometry data that describes the geometric extents of thesurface of the object. Frequently, the image-based data will provide aconvenient and accurate way of representing the object, although nonimage-based data may also be used.

[0038] At block 220 the image-based data is divided or partitioned intosmaller portions. Typically the smaller portions correspond toprimitive-based regions, such as regions defined by triangles in thetriangular mesh. For example, image data of the whole object may bedivided into subsets of image data corresponding to multiple trianglesin the triangle mesh that each describe the appearance within atriangle. In other embodiments partitioning may not be used, althoughpartitioning frequently simplifies subsequent calculations, and allowshigher compression if the smaller portions have a higher degree ofspatial coherency than original set of image-based data, which isfrequently the case.

[0039] At block 230 the plurality of smaller portions are standardized.Standardizing may include normalizing the size of each of a plurality ofsubsets of image-based data to a predetermined size, normalizing a shapeof each of the plurality of subsets of image-based data to apredetermined shape. Standardizing may also include using theimage-based data or the subsets of image-based data to compute resampledsubsets of image-based data that correspond to predeterminedstandardized viewing directions. For example, the size and shapestandardized subsets may be replaced by new subsets that correspond topredetermined standardized viewing directions.

[0040] At block 240 approximate graphical representations thatapproximate the standardized portions are created by selectivelyrepresenting non-redundant information from the standardized portions.This may include removing redundant viewing direction information andredundant surface location information. Creating may include creating afirst data structure that is independent of the viewing direction and asecond data structure that includes a plurality of portions that eachcorrespond to a different viewing direction for multiple primitive-basedregions of the object.

[0041] At block 250 the approximate graphical representations may bestored. Frequently, the approximate graphical representations will beprepared ahead of time to facilitate and accelerate rendering anddistribution of the representations.

[0042] The approximate graphical representations may be data structuresthat include a first portion and a second portion. The second portionmay have view-dependent subportions that each correspond to viewingdirections. The first portion may be combined with each of theview-dependent subportions to generate appearances for the particularviewing direction. These portions may be combined usinghardware-assisted computation, such as multitexturing hardware.

[0043] At block 260 a plurality of approximate graphical representationsare accessed, such as from a memory or from another computer connectedvia a transmission medium like the Internet. Accessing may includeaccessing a first data structure that is independent of the viewingdirection and a second data structure that includes a plurality ofportions that each correspond to a different viewing direction. Thesedata structures may correspond to regions of the surface of the object,such as primitive-defined regions.

[0044] At block 270 a plurality of maps are created based on theapproximate graphical representations. According to one embodiment, themaps may be considered matrices or may be expressed as matrices so thatmatrix operations may be performed on the maps. Some of the maps may bebased on the first data structure that is independent of the viewingdirection while other maps may be based on the second data structure.The maps based on the second data structure may have been configured fora particular viewing direction by selecting the particular subset ofdata from the data structure corresponding to the viewing direction.Creating the plurality of maps may include distorting the first datastructure and the second data structure, such as warping, modulating,translating, or otherwise distorting the data structures.

[0045] At block 280 the maps are combined to generate an appearance forthe object. The maps may be combined, according to certain predeterminedcombinations of the maps, to generate an appearance for the object orfor a primitive-defined region of the object. Combining may includeindividually computing or rendering partial appearances of the objectthat may be combined with other partial appearances to create completeappearances. These complete appearances may be used to display theobject on a display device.

[0046] At block 290 the object or a region of the object is displayed ona display device based on the appearance and the maps. The imagedisplayed may be refined or improved by better approximating theappearance and combining more maps.

[0047] Illustration Of The Method Using Singular Value Decomposition(SVD)

[0048] Without sacrificing generality, and to illustrate the invention,a system and method are described in greater detail for using singularvalue decomposition (SVD) to decompose a reflectance representationknown as a surface light field into an approximate graphicalrepresentation and display the object from multiple viewing directionsusing the approximate representation. The surface light field is afour-dimensional reflectance representation, written asf(r, s, θ, φ),that completely defines the radiance of every point on the surface of anobject in every viewing direction. The two parameters r and s describethe surface location and the two parameters θ and φ describe a viewingdirection. SVD was selected to illustrate the broader concepts of theinvention because SVD is robust, produces near optimal approximationsfor a given amount of compression, allows approximations of variousaccuracies, allows progressive refinement of an image, and facilitatesfast rendering using graphics hardware. Other methods are possible.

[0049] Singular value decomposition decomposes and represents a matrixwith a mathematical identify, the identity may be truncated to create anapproximation. According to SVD, the surface light field representationf(r, s, θ, φ) may be decomposed into a sum of products oflower-dimensional surface maps g_(k) and view maps h_(k), as shown bythe following equation: $\begin{matrix}{{f\left( {r,s,\theta,\varphi} \right)} \approx {\sum\limits_{k = 1}^{K}{{g_{k}\left( {r,s} \right)}{h_{k}\left( {\theta,\varphi} \right)}}}} & (1)\end{matrix}$

[0050] This equation approximates the surface light field function as asum of K products of a first function g_(k)(r,s) and a second functionh_(k)(θ, φ).

[0051] In the equation above, the product terms decrease in importancewith increasing index k. That is, the first product term is the mostimportant, followed by the second, etc. Frequently the above sum ofproduct terms will be truncated to generate an approximation to thesurface light field representation. This representation allows anydesired number of product terms to be included to provide any level ofapproximation of the surface light field. For example, only K productterms may be used, where K is less than the number required for an exactapproximation. Efficient algorithms for computing the first fewsummation terms, which correspond to the largest eigenvalues, are wellknown to those skilled in the art. For example, it may be computed usingMATLAB 6.0, available from The MathWorks of Natick, Mass. Fewer termsare generally needed when objects exhibit a high spatial coherency overthe decomposed region. In many cases only two terms are needed.

[0052] This approach offers a number of advantages. First, the approachallows significant compression without significantly sacrificing imagequality. Second, additional terms may be added to improve theapproximation. This yields very well to progressive encoding, whereinstead of rendering a high quality image slowly a lower quality imageis displayed and then refined. This makes the method amenable tostreaming of data over computer networks, such as the Internet.Additionally, this allows computer programs and applications to refineobjects by increasing K or adding additional terms when it is desirableto do so, or when computer resources are available to do so. Third,since the surface and view maps may be represented as matrix or texturemaps, they may be quickly and efficiently manipulated, such as throughmatrix operations. Additionally, graphics hardware that supportsmultitexturing may form a product of each pair of the surface and viewmaps in comparatively few operations.

[0053] Data Acquisition and Resampling

[0054] Frequently, the surface light field is available in a discreteform as a set of points in a four-dimensional space, rather than in acontinuous form. For example, the surface light field may correspond toa sparse set of images registered to a polygonal approximation (e.g., atriangular mesh) of the surface of the object. In such situations,preprocessing and resampling may be used to standardize the data andfacilitate the system and method. Particular preprocessing is discussedbelow, although other approaches are possible. In other implementations,some or all of preprocessing may not be used, although for the case ofSVD it frequently simplifies the method and gives better results.

[0055] Data Acquisition

[0056] The surface light field may be generated with image-based data,such as acquired geometry and reflectance data. The input data consistof a triangular mesh acquired by a range scanner and pictures taken fromknown locations under constant lighting conditions. A total of N_(I)(typically 150<N_(I)<300) images are captured with a hand-held digitalcamera. An exemplary image may have a 2000×1312 pixel size. However, theimage size is not important to the present invention. In one embodiment,the object is placed on a specific platform designed for the purpose ofautomatic registration. The platform contains indicia, such as coloredcircles, that allow and guide the automatic detection of the gridcorners that are used for computing the 3D location of the camera. Thecolor circles are automatically detected on the images using a singlecolor segmentation scheme. This provides an initial guess for theposition of the grid corners that are then accurately localized using acorner finder. The precise corner locations are then used to compute the3D position of the camera relative to the object. This may be done,given that the camera has been pre-calibrated. The outcome of thisprocess is a set of the N_(I) images captured from known vantage pointsin 3D space. The object geometry is computed in a second stage of dataacquisition. A structured lighting system consisting of a projector anda camera is used for that purpose. The projector is used to project atranslating striped pattern onto the object. Temporal analysis is usedfor accurate range sensing. In order to facilitate scanning, the objectis painted with white removable paint, which is especially useful whendealing with dark, specular, and semi-transparent objects. Typically,10-20 scans are needed, although this number depends on the desiredaccuracy and complexities of the object. Between consecutive scans, theobject is rotated in front of the camera and projector, such as by 20degrees. For that purpose, the calibration platform has been designed torotate about a central axis. The individual scans are automaticallyregistered together in the object reference frame using the same gridcorners previously used for image registration. The complete 3Dtriangular mesh may consist of approximately 7228 triangles. Theresulting cloud of points, which may be approximately 500,000 points, isfed to mesh-editing software to build the final rectangular surfacemesh. Since the same calibration platform is used for image and geometryacquisition, the resulting triangular mesh is naturally registered withrespect to the N_(I) camera images. The geometry and the images may beprecisely aligned, with image reprojection errors of less than onepixel. This image-based data may be stored in a memory.

[0057] The image-based data may be accessed, such as from memory andpreprocessed to facilitate subsequent processing. Preprocessing mayinclude visibility computation, which consists of identifying, for eachmesh triangle, the set of camera images it is visible from. The processbegins by rendering the object from the vantage point of the firstcamera image and reading out the resulting z-buffer, an area in graphicsmemory reserved for storing the Z-axis value of each pixel. The processcontinues by transforming the triangle vertices into the reference frameof the first camera image. The depth of the triangle vertices in thereference frame of the first camera image can then be compared againstthe corresponding values in the z-buffer to determine the visibility ofthe triangle in the first camera image. If at least one of the verticesof the triangle is not visible, the whole triangle is marked asoccluded. This process is repeated for all N_(I) images to give a listof views each triangle is visible from. Thus, the image-based data isdivided or partitioned into smaller portions based on triangle-definedregions of the object. This information is used to resample the rawlight field data.

[0058] Resampling

[0059] Standardization of the smaller portions of the object may beperformed. For example, the size and shape of the triangles may bestandardized and a standardized set of viewing directions may be used torepresent the triangles. During data acquisition the camera views thateach triangle is visible in and the viewing angles of these views in thetriangle reference frame were computed. Note that the visible triangleviews correspond to a set of texture patches of irregular size capturedfrom irregular viewing directions. The texture patches may be resampledinto a more convenient and easier to process format beforedecomposition. First, resampling is performed on the sizes of thetexture patches. Then resampling is performed on the viewing directions.

[0060] Normalization of the sizes of the texture patches may beperformed. As will be explained below, each view of the triangle Δ_(i)consists of exactly the same number of samples M_(Δi). This correspondsto the fact that each column of matrix F^(Δi) has the same size. Thesame is true for matrix F^(vj). To this end, each texture patch isnormalized to a consistent size and shape by warping it using texturemapping graphics hardware. According to one embodiment, the view thatprojects to the largest image area may be identified, this area may bemultiplied by 2, all triangles may be resized to this area, and then thedecomposed light field maps may be subsampled by 2 after thedecomposition. The number of pixels after resampling, M_(Δi), variesfrom 1 pixel, for 1 triangle textures, to 2080 pixels, for 64×64triangle textures.

[0061] Resampling of texture viewing directions may also be performed.Note that the resampling of texture images ensures that there are auniform number of samples for each triangle view, but does not ensurethat sampling of views is uniform. However, a regular sampling of theviewing directions makes the method work better. For example, consider atriangle with a lot of similar views and a few distinct views, thesingular value decomposition of these views will produce a result thatis biased toward the cluster of similar views and that neglects theother views. This can be avoided by resampling the triangle views in thespace of viewing directions before performing the decomposition. Thatis, the viewing direction for the visible triangle views is projectedusing Equation 13, provided below. Then, the weighting factors for theviewing directions that project on a regular grid of points from theDelaunay triangulation of the original projected views are computed.Finally, a set of triangle views that project on the regular grid byblending the initial triangle views using the corresponding weightingfactors are computed. This assumes the viewing direction is constantacross each triangle, which is a good approximation for small triangles,such as those in the mesh. Alternatively, more accurate interpolationsare possible.

[0062] The resolution of the viewing angles can vary from triangle totriangle, however a fixed number of samples is frequently selected. Thecurrent discussion pertains primarily to a resolution of 32×32=1024.Generally there is no visible image degradation when 16×16=256resolutions are used. Choosing fixed numbers of viewing directionsacross separate triangles simplifies the construction of the vertexlight fields F^(vj) since the triangle light field matrices F_(Δi) ^(vj)making up the vertex light field matrix all have the same number ofcolumns.

[0063] The resulting graphical representation of the object may bestored in memory. Typically, these representations are large in size.These representations may be accessed and approximate graphicalrepresentations may be created that contain less redundant informationand that are faster to render.

[0064] Applying Singular Value Decomposition

[0065] The discrete surface light field discussed above is afour-dimensional grid ƒ(r_(p), s_(p), θ_(q), φ_(q)), where p=1, . . . ,M refers to the discrete values (r_(p), s_(p)) describing the locationon the surface of the object and index q=1, . . . , N refers to thediscrete values (θ_(q), φ_(q)) of the viewing angles. Approximations maybe created for each of these surface locations (e.g., triangle portions)by representing non-redundant information. The decomposition accordingto Equation 1 then becomes: $\begin{matrix}{{f\left( {r_{p},s_{p},\theta_{q},\varphi_{q}} \right)} \approx {\sum\limits_{k = 1}^{K}{{g_{k}\left( {r_{p},s_{p}} \right)}{h_{k}\left( {\theta_{q},\varphi_{q}} \right)}}}} & (2)\end{matrix}$

[0066] In other words, the discrete surface light field function isapproximated by a sum of K product terms of surface maps, g_(k), andview maps, h_(k). The surface maps and view maps are collectively knownas light field maps.

[0067] The four-dimensional surface light field data is rearranged intoa matrix. As an example, consider a surface light field ƒ^(P)(r_(p),s_(p), θ_(q), φ_(q)) corresponding to a surface element P. The4-dimensional structure of this partial surface light field isrearranged into a matrix so that each row of the matrix corresponds to adifferent location on the surface element P and each column of thematrix represents a different viewing direction for a given sample. Ifsurface element P consists of M distinct surface samples and eachsurface location is visible from N viewing directions, the resulting M×Nmatrix can be written as:

F ^(P)[ƒ₁ ^(P)ƒ₂ ^(P)

ƒ_(N) ^(P)]  (3)

[0068] Where vector ƒ_(i) ^(P) of size M×1 represents the appearance ofthe surface element P under the i^(th) viewing direction.

[0069] Before discussing the approach in further detail it may behelpful to discuss terminology. A matrix refers to a square orrectangular field of numerical values divided into horizontal rows andvertical columns. An M×N matrix is a matrix with M rows and N columns. Adiagonal matrix is a square matrix having an equal number of rows andcolumns and in which only the elements situated on the main diagonal ofthis matrix (i.e., the first element in the first row, the secondelement in the second row, etc.) has a non-zero value. A transposedmatrix is a matrix formed by interchanging the rows and columns of areference matrix. Frequently a transposed matrix will be denoted by theraised index T (e.g., X^(T)). A vector is a series of discrete numericalvalues that can be regarded as a one-dimensional matrix. Matrix is meantto include vector, since a vector is a special type of matrix.

[0070] The singular value decomposition of F^(P) is the factorizationF^(P)=USV^(T) where the two square matrices U=[u₁, u₂, . . . , u_(M)]and V=[v₁, v₂, . . . , v_(N)] are unitary and S=diag(σ₁, . . . , σ_(N))is a diagonal matrix of positive and monotonically decreasing singularvalues σ_(k). The matrix product USV^(T) can be written as a sum:$\begin{matrix}{F^{P} = {\sum\limits_{k = 1}^{N}{\sigma_{k}u_{k}v_{k}^{T}}}} & (4)\end{matrix}$

[0071] In other words, the matrix F^(P) can be rewritten as a sum of Ndifferent products of σ_(i)u₁ v_(k) ^(v). Truncating the above sum fromN to K corresponds to approximating each column of matrix F^(P) by itsorthogonal projection onto the linear subspace spanned by K vectors u₁,. . . , u_(K). This operation results in an optimal root mean squareapproximation of F^(P) given a fixed budget of K vectors.

[0072] To obtain the form of decomposition as in Equation 2, eachsummation term in Equation 4 is rewritten as (σ_(k) ^(u)u_(k)) (σ_(k)^(v)v_(k)) where σ_(k)=σ_(k) ^(u)σ_(k) ^(v)and rearrange vectoru_(k)′=σ_(k) ^(u)u_(k) into the surface map g_(k) ^(P)(r_(p), s_(p)) andvector v_(k)′ into the view map h_(k) ^(P) (θ_(q), φ_(q)). There arewell documented and efficient algorithms known to those skilled in theart for computing partial SVD giving vectors corresponding to the firstfew largest singular values.

[0073] Those skilled in the art will appreciate that singular valuedecomposition may be applied to the surface light field representationin different ways, without deviating from the spirit and scope of thepresent invention. The discrete function can be directly manipulatedusing SVD. However, the resulting approximation would typically requiremany decomposition terms. Additionally, the decomposition may beexpensive to compute due to the large size of the surface light fieldmatrix. Instead, it is frequently more practical to partition thecomplete surface light field into smaller representations and performthe computations on each part separately. Partitioning the surface lightfield based on primitives is a logical choice for many implementations.This reduces the computational burden, and additionally the smallregions exhibit significant spatial coherency over the region, whichmeans that these small regions can be significantly and accuratelycompressed using singular value decomposition.

[0074] Without loss of generality, two specific approaches are disclosedbelow. Since the surface will typically be represented with a triangularmesh, it is intuitive to partition the light field matrix intoindividual mesh triangles. This approach is called triangle-centeredapproximation and will be discussed first. The second approach is calledvertex-centered approximation and may be used when discontinuities inthe appearance of the object are not desirable.

[0075] Triangle-Centered Approximation

[0076] The discrete representation may be partitioned between eachtriangle in the mesh geometry, as given by the equation: $\begin{matrix}{{f\left( {r,s,\theta,\varphi} \right)} = {\sum\limits_{i = 1}^{n}{{\Pi^{\Delta_{i}}\left( {r,s} \right)}{f\left( {r,s,\theta,\varphi} \right)}}}} & (5)\end{matrix}$

[0077] In other words, the surface light field is rewritten as a sumover the total number of mesh triangles, n, of a product of the triangledomain function Π^(Δ) ^(_(i)) (r, s), which is equal to one within thetriangle Δ_(i) and zero everywhere else, and the surface light field.Each summation term in Equation 5 has then a finite support over asingle triangle and corresponds to the part of the surface light fieldover this triangle.

[0078]FIG. 3 shows a triangle-centered coordinate system or referenceframe for an exemplary triangle 310 belonging to a triangular mesh usedto represent the object. The triangle includes a first vertex v₁, asecond vertex v₂, and a third vertex v₃. The triangle includes an xyzcoordinate system 320 located sufficiently close to the center and wherethe x- and y-axis are in the plane of the triangle and the z-axis isperpendicular to the plane of the triangle. The x-axis is parallel tothe side connecting v₁ and v₂ and the y-axis is parallel to the sideconnecting v₂ and v₃. The triangle is viewed from an eye location 330with a particular viewing direction with respect to the origin of thecoordinate system. A unitary viewing direction vector d points from thecoordinate system toward the eye location 330 and is used to capture theviewing direction. The unitary vector is identified in terms of twoviewing direction angles θ and φ, the azimuth and elevation angles ofthe unitary vector in the fixed reference frame attached to thetriangle. The angle θ is the angle in the xy-plane between the x-axisand a projection of the unitary vector into the xy-plane. Likewise, theangle θ is the angle between the projection and the unitary vectorpointing from the origin of the coordinate system towards the eyelocation. In the final step of partitioning, the surface light field isreparameterized corresponding to each triangle in the local coordinates,to generate a triangle light field, and denoted as ƒ^(Δ) ^(_(l)) (r, s,φ, θ). Note that the same letters are used to denote the localparameters of the triangle light field to simplify the notation.

[0079] Consider the approximation of the discrete surface light fieldover triangle Δ_(I), which is similar to the approximation given byEquation 2, but in the triangle-centered approach: $\begin{matrix}{{f^{\Delta_{i}}\left( {r_{p},s_{p},\theta_{q},\varphi_{q}} \right)} \approx {\sum\limits_{k = 1}^{K}{{g_{k}^{\Delta_{i}}\left( {r_{p},s_{p}} \right)}{h_{k}^{\Delta_{i}}\left( {\theta_{q},\varphi_{q}} \right)}}}} & (6)\end{matrix}$

[0080] where (r_(p), s_(p)) are the M^(Δ1) discrete surface samples oftriangle Δ_(i), and (θ_(q), φ_(q)) are the N discrete viewing directionangles represented in the triangle reference frame. Just as the discretesurface light field data ƒ^(P)(r_(p),s_(p),φ_(q),θ_(q)) are rearrangedcorresponding to the surface element P into matrix F^(P), the discretetriangle light field data ƒ^(Δ) ^(_(l)) (r_(p),s_(p),φ_(q),θ_(q)) isrearranged into a matrix:

F ^(Δ) ^(_(l)) =[ƒ₁ ^(Δ) ^(_(l)) ƒ₂ ^(Δ) ^(_(l))

ƒ_(N) ^(Δ) ^(_(l)) ]  (7)

[0081] where each column of the matrix represents the appearance of thetriangle Δ_(i) under a different viewing direction. The resulting matrixof size M^(Δi)×N is referred to as the triangle light field matrix. Thesame approximation steps that were performed to process matrix F^(P) maybe used to process this matrix to give the approximation correspondingto Equation 6.

[0082] Although the triangle-based decomposition offers an elegant wayof representing and compressing the complete surface light field data,when rendered, it produces visible discontinuities at the edges of thetriangles for a small number of decomposition terms.

[0083] Vertex-Centered Approximation

[0084] The visible discontinuities across triangle boundaries may beeliminated by partitioned the surface light field matrix around everyvertex. This approach is written analogously to Equation 5:$\begin{matrix}{{f\left( {r,s,\theta,\varphi} \right)} = {\sum\limits_{j = 1}^{m}{{\Lambda^{v_{j}}\left( {r,s} \right)}{f\left( {r,s,\theta,\varphi} \right)}}}} & (8)\end{matrix}$

[0085] where m is the total number of vertices v_(j) in the meshgeometry, and each umbrella function Λ^(vj)(r,s) takes values betweenzero and one in the ring of triangles around vertex v_(j), and zeroeverywhere else. Each summation term in Equation 8 has then a finitesupport over the ring of triangles around v_(j) and corresponds to thepart of surface light field over this ring. Equation 8 defines a validpartitioning of the surface light field function only if the sum of allweighting functions Λ_(vj)(r,s) equals 1 over the whole object surface.One possible choice for the functions Λ^(vj)(r,s) is the barycentricweight of each point in the ring of triangles relative to vertex v_(j).Such results are sufficiently continuous for many applications.

[0086]FIG. 4 shows the three umbrella functions Λ^(v1), Λ^(v2), andΛ^(v3) for the three vertices of an exemplary triangle Δ_(i). Theexemplary triangle has vertices v₁, v₂, and v₃. The exemplary triangleis surrounded by ten triangles that are all alike in that each oneshares a vertex with the exemplary triangle. Four separate instances ofthe ten triangles are shown, including a first instance 410 toillustrate the umbrella function for v₁ (labeled Λ^(v1)), a secondinstance 420 to illustrate the umbrella function for v₂ (labeledΛ^(v2)), a third instance 430 to illustrate the umbrella function for v₃(labeled Λ^(v3)), and a fourth instance 440 to show the combination ofthe three umbrella functions (e.g., Λ^(v1)+Λ^(v2)+Λ^(v3)). In each case,gradient shading is used to indicate the domain of the umbrellafunction. That is, the umbrella function corresponding to vertex v₁,Λ^(v1), has finite support only over the six triangles making up vertexv₁, as indicated by the gradient shading. Umbrella functions Λ^(v2), andΛ^(v3) correspond to vertices v₂, and v₃ and have similarcharacteristics. The fourth instance shows that the three umbrellafunctions for the three vertices add to one inside triangle Δ₁. That is,(Λ^(v1)+Λ^(v2)+Λ^(v3))=1 inside the exemplary triangle Δ₁. Thiscondition holds in general for all mesh triangles:

Π^(Δ) ^(_(l)) (r,s)=Λ_(Δ) _(l) ^(v) ^(₁) (r,s)+Λ_(Δ) _(l) ^(v) ^(₂)(r,s)+Λ_(Δ) _(l) ^(v) ^(₃) (r,s)  (9)

[0087] In other words, the triangle domain function introduced inEquation 5 to partition the surface light fields based on triangles, isequal to a sum of the three umbrella functions for each of the threevertices of the given triangle evaluated over the triangle.

[0088]FIG. 5 shows a vertex-centered coordinate system or referenceframe for an exemplary vertex v belonging to a triangular mesh used torepresent the object. The vertex v is a common vertex for each oftriangles Δ₁, Δ₂, Δ₃, Δ₄, and Δ_(R) that form a ring of triangles 510.Although only five triangles are shown, the method is applicable to anynumber of triangles R. The vertex v is the origin of an xyz coordinatesystem 510 where the z-axis is parallel to the surface normal at thevertex and the x-axis and y-axis are mutually perpendicular and in aplane perpendicular to the z-axis. The vertex is viewed from an eyelocation 520 having a particular viewing direction with respect to thevertex. A unitary viewing direction vector d points from the origin ofthe coordinate system toward the eye location 520 and is used to capturethe viewing direction. The unitary vector is identified in terms of twoviewing direction angles θ and φ, the azimuth and elevation angles ofthe unitary vector in the fixed reference frame attached to thetriangle. The angles θ and φ have a similar interpretation as in thetriangle-centered approximation discussed in FIG. 2.

[0089] In the final step of vertex-centered partitioning the surfacelight fields corresponding to each vertex are reparameterized in thelocal coordinates of the vertex to create the vertex light field, whichis denoted by ƒ^(vj)(r, s, θ, φ). Note that the same letters are used todenote the local parameters of the vertex light field in order tosimplify the notation.

[0090] Consider the approximation of the discrete vertex light fielddata over vertex v_(j): $\begin{matrix}{{f^{v_{j}}\left( {r_{p},s_{p},\theta_{q},\varphi_{q}} \right)} \approx {\sum\limits_{k = 1}^{K}{{g_{k}^{v_{j}}\left( {r_{p},s_{p}} \right)}{h_{k}^{v_{j}}\left( {\theta_{q},\varphi_{q}} \right)}}}} & (10)\end{matrix}$

[0091] where (r_(p),s_(p)) are the M^(vj) surface samples covering thewhole ring of triangles, and (θ_(q), φ_(q)) are the N discrete viewingdirection angles represented in the vertex reference frame. Assume,without loss of generality, that triangles Δ_(i), Δ₁, . . . , Δ_(R) formthe ring around the vertex. Then, similarly to the case of thetriangle-centered decomposition, the vertex light field ƒ^(vj)(r_(p),s_(p), φ_(q), θ_(q)) may be represented by the matrix: $\begin{matrix}{F^{v_{j}} = \begin{bmatrix}{w_{\Delta_{i}}^{v_{j}}F^{\Delta_{1}}} \\{w_{\Delta_{2}}^{v_{j}}F^{\Delta_{2}}} \\ \\{w_{\Delta_{R}}^{v_{j}}F^{\Delta_{R}}}\end{bmatrix}} & (11)\end{matrix}$

[0092] where each matrix w_(Δ) _(l) ^(vj)F^(Δ) ^(_(l)) is the weightedtriangle light field matrix corresponding to the ring triangle Δ_(i) andthe square diagonal matrices w_(Δ) _(l) ^(vj) are computed based on thebarycentric coordinates of the points in the ring following thedefinition of U_(Δ) _(l) ^(v) ^(_(j)) . The resulting matrix F^(vj) ofsize M^(vj)×N is referred to as the vertex light field matrix. The sameapproximation steps used on matrix F^(P) may be used on matrix F^(vj) togive the form of approximation of Equation 10.

[0093] Although the terms of the vertex-centered decomposition arecomputed at each vertex, the resulting approximation can be expressedindependently for each triangle Δ_(i) as: $\begin{matrix}{{f^{\Delta_{i}}\left( {r_{p},s_{p},\theta_{q},\varphi_{q}} \right)} = {\sum\limits_{j = 1}^{3}f_{\Delta_{i}}^{v_{j}}}} & (12)\end{matrix}$

[0094] where the index j runs over the three vertices of the giventriangle and ƒ_(Δ) _(l) ^(v) ^(_(j)) describes the part of the vertexlight field corresponding to the ring of triangle Δ₁.

[0095] Note that in the triangle-centered decomposition each triangleshares its three vertex view maps h_(k) ^(v) ^(_(j)) (θ_(q), φ_(q)),j=1, 2, 3, with the neighbor triangles. This insures an approximationthat is continuous across triangles regardless of the number ofapproximation terms K.

[0096] Compression Ratio Affected by Surface Primitives

[0097] The approximation of surface light fields using light field mapscan be thought of as a method of compressing the raw light field datathat takes advantage of the data's spatial coherence. The compressionratio achieved through this approximation is closely related to the sizeof the surface primitives (e.g., mesh triangles) used during thedecomposition. Large mesh triangles results in a higher compressionratio since a smaller number of view maps can represent the complete setof data. However, such mesh triangles are less spatially coherent thansmaller triangles and need more terms to approximate the original datato the same degree. Additionally, large light field matrices arecomputationally expensive to decompose. These effects may be balanced toselect a convenient point where the compression ratio remains high andthe resulting light field approximation produces good results. Forexample, the size of the mesh triangle may be experimentally varied tobalance obtaining a high compression ratio and preserving the visualquality of the results. Frequently this will give several hundred pixelsfor average surface maps. Typically, the effects can be balanced toachieve two orders of magnitude compression and good visual results.

[0098] Compression of the Surface and View Maps

[0099] The surface light field approximations exhibit additionalredundancy not removed by the singular value decomposition and whendesired some of this redundancy may be removed by additionalcompression. For example, the light field maps corresponding to thedifferent surface primitives often look similar if the materialproperties and the lighting conditions across these surface primitivesare the same. Note that this no longer has to be spatial coherence,since the similar light field maps might correspond to non-contiguoussurface elements. To exploit this redundancy and produce even morecompact representations, these light field approximations can be furthercompressed. Since light field maps are represented as 2D images, theymay be compressed using a variety of existing image compressionalgorithms.

[0100] The types of algorithms used for compression differ depending onwhether run-time performance or storage requirements motivate thecompression. For hardware with a limited high bandwidth texture cache,texture swapping is quite expensive and thus rendering performancedeteriorates significantly if the light field maps do not resideentirely in the cache. For this reason, the compressed light field maptextures may be placed entirely in the texture memory cache. However,this may limit the type of compression algorithms used. For example, wemay need to render from the compressed format or perform thedecompression in hardware. Minimizing storage requirements does notconstrain the compression step as much, and allows different compressionalgorithms to be used.

[0101] By way of example, when run-time memory reduction is important,vector quantization may be used to achieve a 10:1 reduction and hardwaretexture compression (e.g., S3TC) may be used to achieve a 6:1 reduction.When storage reduction is important, wavelet or DCT-based compressionmay be used to achieve an 8:1 reduction and entropy encoding may be usedto achieve an additional 2:1 reduction.

[0102] Compression Results

[0103] Surface light fields for five objects with varying complexity andsurface attributes were compressed using singular value decomposition asdescribed above. Considering three summation terms of themean-extracted, vertex-centered approximation, compression ratios forthe five objects ranged from 68:1 to 117:1. The light field mapsthemselves ranged in size from 8.1 MB to 44 MB. Further compression ofthese light field maps using vector quantization and S3TC hardwarecompression gave total compression ratios that ranged from 2106:1 to4167:1. These ratios are calculated with respect to the raw surfacelight fields, neglecting the geometry and texture coordinateinformation, which is generally small compared to the light field maps.

[0104] Rendering Surface Light Fields

[0105] The surface light field approximations may be rendered to displayan object on a display device. The approximations may be accessed,processed by a rendering algorithm, and the object displayed. Accessingmay be from a memory or from a remote location by a transmission medium.

[0106] Rendering Algorithm

[0107] Both the triangle-centered and the vertex-centered surface lightfield approximations result in a separate set of light field maps foreach mesh triangle. This is reflected by Equation 6 for thetriangle-centered approximation and by Equation 12 for thevertex-centered approximation. Rendering from either approximationproceeds by treating each triangle independently. This means that forboth approximations the rendering procedure may be based on only asingle triangle to completely describe the algorithm. The algorithm maybe used to render each triangle one at a time.

[0108] Maps may be created using the surface light field approximations.FIG. 6 shows conversion of a viewing direction into texture coordinatesto determine the pixels of a view map texture H_(k), by orthographicallyprojecting the points of the hemisphere onto an xy-plane that has beenshifted to fit into the traditional texture coordinates range. An xyzreference frame 610, similar to the reference frames previouslydiscussed, is at the center of a square 615 circumscribing a circle 620which both lie in the xy-plane of the coordinate system. The circlecorresponds to perimeter in the xy-plane of an imaginary unit hemispheresitting on the xy-plane and centered at the origin. The points on thehemisphere represent different viewing directions expressed in referenceframe 610, where the viewing direction is represented by the unitaryviewing direction d pointing from the origin toward an eye location 625.A point P corresponds to the orthographic projection of the unitaryvector into the xy-plane. As previously discussed, the unitary vectorcan be expressed in terms of azimuth and elevation angles, θ and φ. At650 the coordinate system and unitary hemisphere are viewed lookingdirectly into the z-axis. As shown, square 615 now corresponds to aregular two-dimensional square grid 655 with x-values ranging from (0,0)to (1,0), y-values ranging from (0,0) to (0,1), and with the origin ofthe coordinate system located at coordinates (0.5, 0.5). Thiscorresponds to the xy-plane shifted to fit into the traditional texturecoordinates range. In this grid point P corresponds to coordinates(x_(P), y_(P)) which may be assigned a numerical value. This approachallows a simple computation of the texture coordinates:

x=(dx+1)/2y=(dy+1)/2  (13)

[0109] In other words, the x coordinate is computed as where drepresents the viewing direction and vectors x and y correspond to theaxis of the local reference frame of the triangle. Other transformationsfrom 3D directions to 2D maps are possible.

[0110]FIG. 7 shows creation of light field maps for triangle-centeredrendering of an exemplary triangle 710. The exemplary triangle has areference frame 715 that is consistent with the triangle-centeredreference frames previously discussed. The triangle has vertices v₁, v₂,and v₃. Each of the three vertices is viewed from a fixed eye location720 Unitary viewing direction vectors d₁ ^(Δ), d₂₁ ^(Δ), and d₃ ^(Δ) areused to represent the viewing direction to each of vertices v₁, v₂, andv₃, respectively. Surface map texture G_(k) ^(Δ) has texture coordinates(s₁ ^(Δ), t₁ ^(Δ)), (s₂ ^(Δ), t₂ ^(Δ)), and (s₃ ^(Δ), t₃ ^(Δ))corresponding to each of the three vertices v₁, v₂, and v₃. View maptexture H_(k) ^(Δ) has texture coordinates (x₁ ^(Δ), y₁ ^(Δ)), (x₂ ^(Δ),y₂ ^(Δ)), and (x₃ ^(Δ), y₃ ^(Δ)), which may be determined by projectingthe unitary vectors onto the shifted xy-plane as discussed in FIG. 5.

[0111] Triangle-Centered Rendering

[0112] This section explains how the triangle-centered approximationintroduced in Equation 6 can be used for light field mapping. Consideran exemplary triangle with light field texture maps G_(k) ^(Δ) and H_(k)^(Δ), k=1, . . . , K. Recall that the triangle-centered partitioning ofthe surface light field introduced a local coordinate system for eachmesh triangle, as was shown in FIG. 2.

[0113] Rendering begins by calculating the view texture coordinates(x_(i) ^(Δ), y_(i) ^(Δ)). This is done by applying Equation 13 tovectors d^(i) ^(Δ) representing the viewing directions for trianglevertices v_(i) expressed in the triangle reference frame. Note that thesurface texture coordinates (s_(i) ^(Δ), t_(i) ^(Δ)) are fixed and donot need to be recalculated when the viewing direction changes.

[0114] The rest of the rendering algorithm proceeds as follows. For thek^(th) approximation term, first render the triangle using surface maptexture G_(k) ^(Δ) by assigning texture coordinates (s_(i) ^(Δ), t_(i)^(Δ)) to vertices v_(i) and store the result. Then render the sametriangle using view map texture H_(k) ^(Δ) by assigning texturecoordinates (x_(i) ^(Δ), y_(i) ^(Δ)) to vertices vi. Performpixel-by-pixel multiplication of the image obtained in the firstrendering pass with the image obtained in the second rendering pass.Note that this corresponds to multiplying pixel-by-pixel the imageprojections of the shaded triangular texture fragment of the surface maptexture and the shaded triangular texture fragment of the view maptexture shown in FIG. 7. The same operations may be used to computeadditional product terms in the summation. Summing all of the productterms generates the final approximation for the particular triangle.This may be repeated for all the visible triangles in the triangularmesh.

[0115] Vertex-Centered Rendering

[0116] This section explains how the vertex-centered approximationintroduced in Equation (12) can be used for light field mapping.Rendering the vertex-centered approximation is very similar to renderingthe triangle-centered approximation. One notable difference is that eachtriangle light field is approximated by a blend of three light fieldmaps for each approximation term, rather than one pair. That is,approximating an exemplary triangle with light field texture maps usesG_(k) ^(v) ^(_(j)) and H_(k) ^(v) ^(_(j)) , k=1, . . . , K and j=1, 2, 3to represent the K terms of the triangle. As was shown and discussed inFIG. 5, the textures are based on a vertex-centered partitioningcoordinate system that uses unique coordinate systems for each vertex ofthe triangular mesh.

[0117]FIG. 8 shows creation of the three pairs of surface and view maptextures for a one-term approximation of the surface light field for anexemplary triangle 810. The triangle has vertices v₁, v₂, and v₃. Afirst depiction 805 shows the triangle with a reference frame 810attached to the vertex v₁. The three unitary viewing direction vectorsd₁ ^(v) ^(₁) , d₂ ^(v) ^(₁) , and d₃ ^(v) ^(₁) , which correspond tovertices v₁, v₂, and v₃, are expressed in the reference frame 810. Apair of surface and view map textures 815, corresponding to the triangleand the unitary vectors expressed in the reference frame 810, may bedetermined substantially as previously discussed. The surface maptextures are weighted based on their barycentric coordinates withrespect to vertex v₁, as indicated by the gradient shading. As indicatedin Equation 11, this is the result of weighting that was applied duringthe construction of matrix F^(v) ^(_(j)) . The weighting did notsignificantly alter the rendering algorithm. A second depiction 820shows the triangle with a reference frame 825 attached to the vertex v₂.The three unitary viewing direction vectors d₁ ^(v) ^(₂) , d₂ ^(v) ^(₂), and d₃ ^(v) ^(₂) , which correspond to vertices v₁, v₂, and v₃, areexpressed in the reference frame 825. A pair of surface and view maptextures 830, corresponding to the triangle and the unitary vectorsexpressed in the reference frame 825. A third depiction shows thetriangle with a reference frame 840 attached to the vertex v₃. The threeunitary viewing direction vectors d₁ ^(v) ^(₃) , d₂ ^(v) ^(₃) , and d₃^(v) ^(₃) , which correspond to vertices v₁, v₂, and v₃, are expressedin the reference frame 840. A pair of surface and view map textures 845,corresponding to the triangle and the unitary vectors expressed in thereference frame 835.

[0118] The rendering algorithm starts by calculating the view texturecoordinates (x_(i) ^(v) ^(_(j)) , y_(i) ^(v) ^(_(j)) ) where irepresents vertex v_(i) and the superscript v_(j) denotes the term isexpressed in the reference frame of vertex v_(j). This is done by usingEquations 13 to vectors d_(i) ^(v) ^(_(j)) representing the viewingdirections for triangle vertices v_(i) expressed in the reference frameof vertex v_(j). Note that the surface texture coordinates (s_(i) ^(v)^(_(j)) , t_(i) ^(v) ^(_(j)) ) are fixed and do not need to berecomputed. In fact, they are the same for each vertex reference frame.

[0119] For the k_(th) approximation term and vertex coordinates v_(j),first the triangle is rendered using surface map texture G_(k) ^(v)^(_(j)) by assigning texture coordinates (s_(i) ^(v) ^(_(j)) , t_(i)^(v) ^(_(j)) ) to vertices v_(i) and the result is stored. Then the sametriangle is rendered using view map texture H_(k) ^(v) ^(_(j)) byassigning texture coordinates (x_(i) ^(v) ^(_(j)) , y_(i) ^(v) ^(_(j)) )to vertices vi. Pixel-by-pixel multiplication of the image obtained inthe first rendering pass with the image obtained in the second renderingpass is performed and the result is stored. Note that the above processcorresponds to multiplying pixel-by-pixel the image projections of thetwo shaded texture fragments shown in each row of FIG. 8. The results ofeach multiplication of the two textures are to be added together and therendering routine is repeated for each mesh triangle.

[0120] Hardware Accelerated Rendering

[0121] The predetermined combinations of maps discussed above may becombined to generate an appearance for the object. Computing orrendering of the appearances may be performed using traditionalrendering methods and hardware. The traditional pixel pipeline involveseight stages: (1) stipple test, (2) z-buffer, (3) fetch texels fromsingle texture; filter to final texel color, (4) texture blend, (5) addspecular, (6) apply fog, (7) alpha blend, and (8) write to frame buffer.Combining two textures using traditional rendering methods and hardwareinvolves rendering by applying texture mapping to the surface map,writing to the accumulation buffer, applying texture mapping to the viewmap, combining it with the contents of the accumulation buffer bypixel-by-pixel multiplication of the result of the first rendering bythe result of the second rendering, and transferring the contents of theaccumulation buffer into the frame buffer.

[0122] Alternatively, computing or rendering the appearances may beperformed using hardware accelerated rendering, such as by using agraphics accelerator. The graphics accelerator may have fast memory(e.g., DRAM, VRAM), a video bus, and one or more graphics processors forfast and efficient execution of graphics related calculations, such asmatrix operations. The graphics accelerator may receive tasks from themain CPU and carry out those tasks.

[0123] The particular approximation through the decomposition into thelight field maps using SVD yields well to hardware accelerated renderingwith multitexturing hardware. Multitexturing allows objects to bedisplayed with more texturally enhanced graphics to be used in about thesame amount of computer time. Multitexturing hardware allows multipletextures to be manipulated in the same rendering pass, which increasesoverall rendering performance. For example, a first texture, such as abrick wall, may be combined with a second texture, such as graphiti.

[0124] Multitexturing allows multiple textures to be manipulatedsimultaneously. Multitexturing uses multiple hardware units, known astexture operation units, cascaded together to apply multiple textures toa primitive in a single pass. A single texture is associated with eachunit. The texture operation units have render states that control pixelflow through the texture operation unit. For example, it may have sixrender states, with three associated with RGB (color), and anotherassociated with alpha. The texture operation units also have filtering,clamping, and other render states. In a process known as textureblending cascade, results from earlier stages are carried over tosubsequent stages until the final result is rasterized on the polygon.For example, multitexturing renders the geometrical primitive byapplying texture mapping of the surface map to the object using theframe buffer, and subsequently rendering the primitive by applyingtexture mapping of the view map by performing pixel-by-pixelmultiplication with the contents of the frame buffer. This may avoid theaccumulation buffer, which is frequently implemented in software, andcan be a bottleneck.

[0125] Multitexturing hardware allows multiple textures to bemanipulated in the same rendering pass, which makes the hardware verycompatible with certain embodiments of the present invention.Accordingly, representing the surface light fields with light field mapsis very convenient, since these light field maps may be treated astextures by multitexturing hardware. This provides very fast andefficient rendering. Recall that one of the fundamental operations ofthe proposed rendering algorithms was the pixel-by-pixel multiplicationor modulation of the one or more surface texture fragments by thecorresponding one or more view texture fragments. This operationcorresponds to evaluating one term of the approximation sum. Thecomplete sum is evaluated by blending the results of individualmodulation operations. Multitexturing hardware support enables themodulation of the two texture fragments to be computed very efficiently,such as in one rendering pass for certain implementations. This meansthat rendering from the triangle-centered representation requires only Krendering passes, where K is the number of terms in the approximationsum, and the vertex-based rendering algorithm requires 3K renderingpasses to evaluate K approximation terms. Without the multitexturinghardware the approximation can be implemented using the accumulationbuffer, although much less efficiently. After generating the appearance,the object may be displayed by applying the appearance to the geometryof the object and displaying the object on a display device.

[0126] A number of existing consumer-level graphics hardware areavailable to perform multitexturing, including among others, Microsoft®DirectX® 7 available from Microsoft of Redmond, Wash.; and GeForceUltra™ available from nVIDIA of Santa Clara, Calif. As will be explainedbelow, this allows graphics associated with the invention to be renderedquickly. nVIDIA's GeForce Ultra™ provides multitexturing and registercombiner extensions that are very useful for efficient implementation oflight field mapping.

[0127] Light field maps may have negative values. Some hardware offersextended color range. When there are negative values, more renderingpasses may be needed, unless the graphics hardware allows modificationof the limits of the pixel range from the traditional [0,1] range to a[min, max] range, such as [−1.5, 5]. nVIDIA's cards support extendedcolor range through the register combiner extension, but they also clampthe output to positive values. This means that the surface light fieldapproximations can be rendered using register combiners, but at theexpense of an increased number of rendering passes. Let M be the resultcontaining negative values of modulation of two texture fragments A andB. Let M₊ be the result of modulating fragments A and B using theregister combiner extension. Note that M₊ is equal to M where M ispositive and it is zero otherwise. Let M⁻ be the result of modulating −Aand B the same way. Note that M− is equal to −M where the latter ispositive and it is zero otherwise. Therefore, M can be calculated bysubtracting the outputs of the two modulations using blending: M=M₊−M⁻.Since each modulation operation can be performed in one rendering pass,the register combiners extension allows one summation term of thesurface light field approximation that contains negative values to beevaluated using at most two rendering passes.

[0128] Some hardware offers a fixed color range. Surface light fieldapproximations may be rendered even when graphics hardware does notsupport extended pixel range, although the number of rendering passesincreases significantly. In this situation, the texture map fragments Aand B are separated into strictly non-negative fragments A+, B+, A−, B−such that A=A+−A− and B=B+−B−. Then the multiplication of AB iscalculated in four rendering passes of strictly positive terms:AB=A+B+−A−B+−A+B−+A−B−. Blending that supports subtraction can be used.

[0129] Mean extraction may be performed. It is often advantageous tocompute for each triangle the mean triangle view and subtract this fromthe light field matrix before the matrix decomposition. This changes therendering routine slightly. First the triangle is texture mapped usingits mean view, and then the approximation terms of the modified lightfield matrix are blended as previously described. In a lot of cases themean triangle view represents an approximation to the diffuse componentof the surface material and as such is often interesting to visualizeindependently from the light field data. Additionally, rendering themean view triangle and K term approximation of the light field matrixwith the mean view extracted produces comparable results as renderingK+1 term approximation of the original light field matrix. This is truefor both the triangle-centered and the vertex-centered approximations ofthe light field data. Recall that each term of the vertex-centeredapproximation consists of a blend of 3 vertex light fields and, in thebest case, requires 3 rendering passes to compute. Using mean viewtexture can therefore reduce the number of rendering passes for thevertex-centered approximation.

[0130] If rendering performance is of utmost importance, positivedecomposition may be used. The light field matrix approximations can bemodified so that computation can be performed with a minimal number ofrendering passes. Recall that modulation of two texture fragments A andB produces a strictly positive result that can be rendered in onerendering pass using multitexturing hardware support. This can be usedto reduce the rendering time of the light field approximations. Considerthe triangle light field matrix F^(Δi) in Equation 7 and let vectorƒ_(m) ^(Δ) ^(_(l)) store the minimum value in each row of this matrix.Note that the matrix F₊ ^(Δ) ^(_(l)) , obtained from subtracting vectorƒ_(m) ^(Δ) ^(_(l)) from each column of matrix F^(Δi) has no negativeelements. Since the singular value decomposition process has a propertythat the singular vectors corresponding to the largest singular value ofthe positive matrix are also positive, the light field mapscorresponding to the first term of the decomposition of matrix F₊ ^(Δ)^(_(l)) will be strictly positive. Therefore, if the matrix F^(Δi) isapproximated as a sum of vector ƒ_(m) ^(Δ) ^(_(l)) and the firstdecomposition term of matrix F₊ ^(Δ) ^(_(l)) this approximation can bedisplayed in just two rendering passes, one rendering pass to displayeach term of the sum. Note that the vertex light field matrix F^(vj) inEquation 11 can be decomposed in the same way, and that the resultingapproximation can be evaluated in four rendering passes. For eachtriangle, one rendering pass is used to display the minimum vector andthree rendering passes are used to display the positive decompositionterm of each of the triangle vertices.

[0131] Multitexturing hardware may also have capabilities to allow morethan two textures to be combined simultaneously. For example,multitexturing hardware may allow four or more textures to be accessedin one rendering pass. This extra capacity can be used in different waysto increase rendering speed and efficiency. Typically, it will be usedto add more summation terms to the approximation. However, it may alsobe used to combine approximation terms for other triangles. Therendering operations may need modification to do this.

[0132] Rendering Results

[0133] Surface light field approximations for four objects with varyingcomplexity and surface attributes were rendered using nVIDIA's GeForceUltra™ graphics card on a PC-class hardware. Considering thevertex-centered rendering technique, one term approximations used 7rendering passes, 2 terms used 13 rendering passes, and 3 terms used 19rendering passes. Rendering performance varied between 22.4-32.7frames-per-second (FPS) for the one-term approximations, 13.5-20.4 FPSfor the two-term approximations, and 10.1-14.1 FPS for the three-termapproximations. In each case, the results were displayed in a 1K×1Kwindow with objects occupying approximately ⅓ of the window dimensions.The rendering performance is not very sensitive to image size. When theimage size doubled, the frame rate decreased by only about 20%.

[0134] Providing Graphical Content Over a Transmission Medium

[0135]FIG. 9 shows an exemplary system 900 for delivering graphicalcontent over a transmission medium, such as the Internet. Although anInternet transmission medium is described, other transmission mediumsare possible, such as other computer networks (e.g., LANs, WANs, MANs),wireless communication devices such as pagers and phones, and otherdevices.

[0136] A first computer system 910 includes an Internet interface 915 toallow the first computer system to transmit graphical content via theInternet. The Internet comprises an Internet interconnectivity 950 thatmay include routers, switches, and the like to allow transmission ofdata. A second computer system 960 includes an Internet interface 965 toallow the second computer system to receive the graphical content viathe Internet.

[0137] The first computer system typically includes a collection system920 to input a graphical representation of an object 925. The collectionsystem may include cameras, range scanners, and the like. Alternatively,the collection system and the object may not be used if the graphicalrepresentation is loaded from a computer-readable medium, or createdusing software. Typically, the first computer system will includecomponents 930, typically software components, to acquire, process, andcompress the graphical content that is to be transmitted. The componentsmay include a data acquisition manager 932 to assist with acquiringgraphical data for an object, such as from the collection system. Apreprocessing manager 934 may be used to preprocess, resample, andstandardize the graphical data prior to approximation. An approximationmanager 936 may be used to create an approximate graphicalrepresentation based on the graphical data. A compression manager may beused to perform traditional compression on the approximate graphicalrepresentations. Naturally, there will be other components, such ascomponents to enable the described components, and components to allowefficient interaction with the second computer system. For example, acomponent may be provided to determine how many approximation terms totransmit, by either requesting the information from the client, using apredetermined number for a certain speed of connection (e.g., moreapproximation terms for faster connections), monitoring congestion, andothers. Typically, the approximate graphical representations will beprepared ahead of time and stored in a graphical representations library935, which may be a memory or other computer-readable medium.

[0138] The first computer system transmits the approximate graphicalrepresentations and instructions that assist with interpreting and usingthe approximate graphical representations to the second computer systemvia the Internet interface. This will typically include transmitting aviewer, such as a plugin to the second computer system's web browser asan add-on application. This may be transmitted during this session, orduring a previous or subsequent session. It may be registered with thebrowser, so that it can be invoked when the browser receives or detectsan incoming file format or data structure associated with the viewer.Once activated, the viewer may perform as a stand-alone application. Forexample, the viewer may use instructions that allow the second computersystem to access the file format or data structure of the compresseddata, interpret it in a way that is consistent with the method used tocompress it, and use the data to display a representation of the object,such as on a display device. The instructions may allow multiprocessing,parallel processing, hardware accelerated rendering, or multitexturingto be used to display the object. According to one embodiment,instructions are available for multiple types of graphical hardware,such as multitexturing hardware from different hardware vendors. Thesecond computer system may be examined to determine the appropriateinstructions to use. Alternatively, the second computer system mayreceive instructions specific to the particular hardware at the secondcomputer system. The viewer may allow the first computer system todisplay the graphical content automatically with little additionalinteraction from the user of the first computer system.

[0139] The second computer system typically includes a display device970 to display the graphical representation, a data entry device 975,such as a keyboard or a curser control device, to allow data entry orspecification of user preferences. The second computer system may alsohave graphics hardware 980, which may be multitexturing hardware withmultiple texture operation units 985. The second computer system willtypically include components 990, such as software components that maybe provided by the viewer, to provide a useage interface, manage usageof the transmitted graphical data, and perform rendering based on thetransmitted graphical data. The components may include a user interfacemanager 992, such as a graphical user interface, to allow the user toeasily interact with the approximate graphical representation and thefirst computer system. An interpretation manager 994 may interpret theapproximate graphical representation and operate on the representationto put it in a format that may be rendered. For example, considering thesingular value decomposition presented above, the approximationgenerator manager may determine an appropriate portion of the view mapsto use in rendering, based on the viewing direction. The interpretationmanager may also perform decompression according to vector quantization,or other standard compression techniques if the light field maps havebeen compressed by these standard techniques. The rendering manager 996assists with displaying the graphical representation of the object, suchas on the display device. For example, the rendering manager may usemultitexturing hardware to combine surface and view texture maps for aparticular viewing direction, or may provide software to mimic theoperations of this hardware. These components may perform additionaloperations, such as any of the operations discussed above for theviewer, or other components may be used to add functionality.

[0140] Consider an exemplary use of the system in which a user of thesecond computer system uses a browser to access a web page associatedwith the first computer system. The web page has graphical content (orlinks to graphical content) associated with approximate graphicalrepresentations in accordance with the invention. The first computersystem transmits approximate graphical representations, such as datastructures in particular formats in accordance with various embodimentsof the present invention, to the second computer system. For example, itmay transmit a geometry representation, such as a triangular mesh, and areflectance representation, such as surface and view maps. The secondcomputer system may use a viewer downloaded previously from theInternet, to use the transmitted graphical data to display the graphicalcontent. The user may then use the viewer, which may provide functionalbuttons, dials, data entry locations, etc, and the data entry device tochange a viewing direction, viewing distance, select another object,such as by selecting the object from a list, or search a remote databasefor another object. Other uses will be apparent to those skilled in theart.

[0141] Exemplary Computer Architecture

[0142] As discussed herein, a “system”, such as a system forapproximating a graphical representation, may be an apparatus includinghardware and/or software for processing data. The system may include,but is not limited to, a computer (e.g., portable, laptop, desktop,server, mainframe, etc.), hard copy equipment (e.g., film recorder,optical disk burner, printer, plotter, fax machine, etc.), and the like.

[0143] A computer system 1000 representing an exemplary workstation,host, or server in which features of the present invention may beimplemented will now be described with reference to FIG. 10. Thecomputer system 1000 represents one possible computer system forimplementing embodiments of the present invention, however othercomputer systems and variations of the computer system 1000 are alsopossible. The computer system 1000 comprises a bus or othercommunication means 1001 for communicating information, and a processingmeans such as processor 1002 coupled with the bus 1001 for processinginformation. The computer system 1000 further comprises a random accessmemory (RAM) or other dynamic storage device 1004 (referred to as mainmemory), coupled to the bus 1001 for storing information andinstructions to be executed by the processor 1002. The main memory 1004also may be used for storing temporary variables or other intermediateinformation during execution of instructions by the processor 1002. Inone embodiment, the main memory 1004 may be used for storing graphicalrepresentations, compression and approximation instructions, approximategraphical representations, and rendering instructions. The computersystem 1000 also comprises a read only memory (ROM) and other staticstorage devices 1006 coupled to the bus 1001 for storing staticinformation and instructions for the processor 1002. A data storagedevice 1007 such as a magnetic disk, zip, or optical disc and itscorresponding drive may also be coupled to the computer system 1000 forstoring information and instructions. In one embodiment, the datastorage device 1007 may be used to access graphical representations andstore approximate graphical representations to film, magnetic, optical,or other media.

[0144] The computer system 1000 may also be coupled via the bus 1001 toa display device 1021, such as a cathode ray tube (CRT) or LiquidCrystal Display (LCD), for displaying information to an end user.Typically, a data input device 1022, such as a keyboard or otheralphanumeric input device including alphanumeric and other keys, may becoupled to the bus 1001 for communicating information and commandselections to the processor 1002. Another type of user input device is acursor control device 1023, such as a mouse, a trackball, or cursordirection keys for communicating direction information and commandselections to the processor 1002 and for controlling cursor movement onthe display 1021.

[0145] A communication device 1025 is also coupled to the bus 1001.Depending upon the particular implementation, the communication device1025 may include a modem, a network interface card, or other well-knowninterface devices, such as those used for coupling to Ethernet, tokenring, or other types of physical attachment for purposes of providing acommunication link to support a local or wide area network, for example.In any event, in this manner, the computer system 1000 may be coupled toa number of clients or servers via a conventional networkinfrastructure, such as a company's intranet or the Internet, forexample. The communication device may be used to send graphicalrepresentations to other computer systems.

[0146] The invention is not limited to any particular computer system.Rather, embodiments may be used on any stand alone, distributed,networked, or other type of computer system. For example, embodimentsmay be used on one or more computers compatible with NT, Linux, Windows,Macintosh, any variation of Unix, or others.

[0147] The present invention includes various steps, as described above.The steps of the present invention may be performed by hardwarecomponents or may be embodied in machine-executable instructions, whichmay be used to cause a general-purpose or special-purpose processor orlogic circuits programmed with the instructions to perform the steps.Alternatively, the steps may be performed by a combination of hardwareand software.

[0148] The present invention may be provided as a computer programproduct that may include a machine-readable medium having stored thereoninstructions that may be used to program a computer (or other electronicdevices) to perform a process according to the present invention. Themachine-readable medium may include, but is not limited to, floppydiskettes, optical disks, CD-ROMs, and magneto-optical disks, ROMs,RAMs, EPROMs, EEPROMs, magnet or optical cards, flash memory, or othertype of media or machine-readable medium suitable for storing electronicinstructions. Moreover, the present invention may also be downloaded asa computer program product, wherein the program may be transferred froma remote computer to a requesting computer by way of data signalsembodied in a carrier wave or other propagation medium via acommunication link (e.g., a modem or network connection).

[0149] Alternative Embodiments of the Present Invention

[0150] Those skilled in the art will recognize that many variations ofthe present invention are possible. The appearance representation to beapproximated may be pre-filtered to reduce noise or redundancy beforeapplying the method. This may be useful if the filtering has a higherselectivity for important information than the invention, but does nothave other features of the invention.

[0151] The surface light field representation could also be converted toanother color space. For example, the surface light field matrix couldbe mathematically converted to YUV color space, which is based onluminance and chromanance. This may allow more chrominance informationto be discarded, especially at higher frequencies, since thisinformation is typically is less important to human perception anyway.

[0152] Additional parameters may be added to the functions described.For example, the method may be applied to functions where additionalparameters have been added to allow various features to change in time.For example, additional parameters may allow the location of the lightsource to be varied, the intensity of the light to be varied, and otherflexibilities. Modified methods will be apparent to those of skill inthe art, based on the disclosure herein.

[0153] Also, many uses of the present invention will be apparent tothose skilled in the art. For example, the invention may be used toproviding visual representations of objects in conjunction with sellingthose objects, such as on the Internet or otherwise, may be used toimprove video games, may be incorporated in computer generated imagessuch as to add special effects to feature films. For example, agraphical representation may be used to add a spaceship to a film scene,where the spaceship is viewed from different viewing directions underconstant lighting conditions, such as those provided by a nearby star.The invention may also be used to add content to 3D TV or HDTV. Or thegraphical representations may be included in machine-readable format,such as on a CD-ROM, in the form of a famous sculpture distributed aspart of an electronic encyclopedia, and other uses. Other uses will beapparent.

[0154] In conclusion, the present invention provides a system and methodfor compressing graphical data for a three-dimensional object.Compressing includes approximating a graphical representation thatdescribes the appearance of the object from multiple viewing directionsby removing redundant information from the graphical representation. Thesystem and method may allow the graphical data to be displayed usinghardware-assisted computation.

[0155] In the foregoing specification, the invention has been describedwith reference to specific embodiments thereof. It will, however, beevident that various modifications and changes may be made theretowithout departing from the broader spirit and scope of the invention.The specification and drawings are, accordingly, to be regarded in anillustrative rather than a restrictive sense.

What is claimed is:
 1. A method comprising: accessing a firstmulti-dimensional graphical representation that describes the appearanceof a plurality of points of an object from a plurality of viewingdirections, the appearance varying from point to point and viewingdirection to viewing direction; creating a second graphicalrepresentation that approximates the first graphical representation andthat includes at least one expression having a fewer dimensions than thefirst multi-dimensional graphical representation by decomposing thefirst multi-dimensional graphical representation into the secondgraphical representation, the decomposing including selectivelyrepresenting information from the first graphical representation.
 2. Themethod of claim 1, further comprising expressing the first graphicalrepresentation as a matrix, and wherein creating the second graphicalrepresentation includes using singular value decomposition to decomposethe first multi-dimensional graphical representation into a first vectormultiplying a second vector.
 3. The method of claim 1, wherein creatingby decomposing includes selectively representing information from thefirst graphical representation that is most important to describe theappearance of the object and neglecting redundant information from thefirst graphical representation that is less important to describe theappearance of the object.
 4. The method of claim 1, further comprising:accessing a third multi-dimensional graphical representation thatdescribes the appearance of a second plurality of points of the objectfrom a plurality of viewing directions, the second plurality of pointsincluding the first plurality of points, and the appearance varying frompoint to point and viewing direction to viewing direction; andpartitioning the third multi-dimensional graphical representation into aplurality of smaller multi-dimensional graphical representations eachassociated with a primitive of a polygonal representation of thegeometry of the object, the plurality of smaller multi-dimensionalgraphical representations including the first multi-dimensionalgraphical representation.
 5. A method comprising: accessing a firstgraphical representation that describes the appearance of a plurality ofpoints on an object; creating a second graphical representation based onthe first graphical representation, the second graphical representationcontaining less redundant descriptive information than the firstgraphical representation, and the second graphical representationcontaining a plurality of portions that are capable of beingconcurrently combined to display the plurality of points of the objectwith an appearance associated with a particular viewing direction. 6.The method of claim 5, wherein the first graphical representationdescribes the appearance of a plurality of points of an object from aplurality of viewing directions, the appearance varying from point topoint and viewing direction to viewing direction.
 7. The method of claim5, wherein creating includes creating a second graphical representationthat includes a plurality of portions capable of being expressed asmatrices, the method further comprising expressing the plurality ofportions as matrices.
 8. The method of claim 5, further comprising:storing the second graphical representation; receiving a request on anetwork for the second graphical representation; transmitting the secondgraphical representation on the network after receiving the request. 9.The method of claim 5, wherein creating includes creating a secondgraphical representation that includes a plurality of portions that canbe combined using multitexturing hardware that allows multiple texturesto be concurrently applied to a primitive in a single rendering pass.10. The method of claim 5, wherein creating includes creating a secondgraphical representation that includes a plurality of portions that arecapable of being concurrently combined without decompressing theplurality of portions.
 11. The method of claim 5, further comprising:accessing a third graphical representation that describes the appearanceof a second plurality of points of the object from a plurality ofviewing directions, the second plurality of points including the firstplurality of points, and the appearance varying from point to point andviewing direction to viewing direction; and partitioning the thirdgraphical representation into a plurality of smaller graphicalrepresentations each associated with at least one primitive of apolygonal representation of the geometry of the object, the plurality ofsmaller multi-dimensional graphical representations including the firstmulti-dimensional graphical representation.
 12. A machine-readablemedium having stored thereon data representing sequences of instructionsthat when executed cause a machine to perform operations comprising:accessing a first graphical representation that describes the appearanceof a plurality of points on an object; creating a second graphicalrepresentation based on the first graphical representation, the secondgraphical representation containing less redundant descriptiveinformation than the first graphical representation, and the secondgraphical representation containing a plurality of portions that arecapable of being concurrently combined to display the plurality ofpoints of the object with an appearance associated with a particularviewing direction.
 13. The machine-readable medium of claim 12, whereinthe instructions for accessing the first graphical representationfurther comprise instructions causing the machine to perform operationscomprising accessing a first graphical representation that describes theappearance of a plurality of points of an object from a plurality ofviewing directions, the appearance varying from point to point andviewing direction to viewing direction.
 14. The machine-readable mediumof claim 12, wherein the instructions for creating further compriseinstructions causing the machine to perform operations comprisingcreating a second graphical representation that includes a plurality ofportions capable of being expressed as matrices.
 15. Themachine-readable medium of claim 12, wherein the instructions forcreating further comprise instructions causing the machine to performoperations comprising creating a second graphical representation thatincludes a plurality of portions that can be combined usingmultitexturing hardware that allows multiple textures to be concurrentlyapplied to a primitive in a single rendering pass.
 16. Themachine-readable medium of claim 12, wherein the instructions forcreating further comprise instructions causing the machine to performoperations comprising creating a second graphical representation thatincludes a plurality of portions that are capable of being concurrentlycombined without decompressing the plurality of portions.
 17. Themachine readable medium of claim 12, wherein the instructions furthercomprise instructions causing the machine to perform operationscomprising: accessing a third graphical representation that describesthe appearance of a second plurality of points of the object from aplurality of viewing directions, the second plurality of pointsincluding the first plurality of points, and the appearance varying frompoint to point and viewing direction to viewing direction; andpartitioning the third graphical representation into a plurality ofsmaller graphical representations each associated with at least oneprimitive of a polygonal representation of the geometry of the object,the plurality of smaller multi-dimensional graphical representationsincluding the first multi-dimensional graphical representation.
 18. Amethod comprising: accessing image-based data for an object thatdescribes the appearance of the object from a plurality of viewingdirections; dividing the image-based data into a plurality of smallerportions associated with regions on the object; standardizing each ofthe plurality of smaller portions; creating a plurality of approximategraphical representations that approximate the plurality of standardizedportions by selectively representing certain non-redundant informationfrom each of the plurality of standardized portions; and storing each ofthe plurality of approximate graphical representations.
 19. The methodof claim 18: wherein dividing the image-based data includes partitioningthe image-based data into a plurality of subsets of image-based data,each subset of image-based data describing the appearance of aprimitive-defined region of the object for a particular viewingdirection, each subset having a higher degree of spatial coherency thanthe set of image-based data; wherein standardizing includes normalizingthe size of each of the plurality of subsets of image-based data to apredetermined size, normalizing the shape of each of the plurality ofsubsets of image-based data to a predetermined shape, and using theplurality of subsets of image-based data to compute a resampledplurality of subsets of image-based data that correspond topredetermined standardized viewing directions; and wherein creating aplurality of approximate graphical representations includes creating foreach primitive-defined region a first data structure that is independentof the viewing direction and a second data structure that includes aplurality of portions that each correspond to a different viewingdirection.
 20. The method of claim 18, further comprising: acquiringgeometry data for an object that describes the geometric extents of thesurface of the object; converting the geometry data into a geometricrepresentation of the geometry of the object; and acquiring image-baseddata that describes the appearance of the surface of the object from aplurality of viewing directions.
 21. The method of claim 18, furthercomprising: receiving a request for graphical content associated withthe object from another computer system; transmitting a plurality ofapproximate graphical representations to the other computer system; andtransmitting geometry data for the object to the other computer system.22. A data structure comprising at least a first portion and a secondportion, the second portion including a second plurality ofview-dependent subportions including a first view-dependent subportionthat corresponds to a first viewing direction and a secondview-dependent subportion that corresponds to a second viewingdirection, wherein the first portion and the first view-dependentsubportion are combinable using multitexturing hardware that allowsmultiple textures to be concurrently applied to a primitive in a singlerendering pass to display a plurality of points of an object with afirst appearance corresponding to a first viewing direction, and whereinthe first portion and the second view-dependent subportion arecombinable using the multitexturing hardware to display the plurality ofpoints of the object with a second appearance corresponding to a secondviewing direction.
 23. The data structure of claim 22, wherein the datastructure is derived from a plurality of images acquired for the objectby selectively representing information from the plurality of imagesthat is important to describe the appearance of the object andselectively removing information from the plurality of images that isredundant.
 24. The data structure of claim 22, wherein the first portionand the first view-dependent subportion are combinable withoutdecompression using the multitexturing hardware.
 25. The data structureof claim 22, wherein the first portion and the second portion arecreated by decompressing a corresponding compressed first portion and acorresponding compressed second portion of another data structure.
 26. Adata structure comprising a plurality of portions that are combinablewithout interpolation to display a plurality of points of the objectwith a first appearance for a first viewing direction and display theplurality of points with a second appearance for a second viewingdirection.
 27. The data structure of claim 26, the data structure beingderived from a plurality of images acquired for an object byrepresenting important information that describes the appearance of theobject and removing redundant information associated with the pluralityof images.
 28. The data structure of claim 26, wherein the plurality ofportions are combinable without decompression.
 29. The data structure ofclaim 26, wherein the plurality of portions are created by decompressingcorresponding compressed portions of another data structure.
 30. Thedata structure of claim 26, wherein the plurality of portions include afirst matrix and a second matrix that are concurrently combinable usingmultitexturing hardware that allows multiple textures to be concurrentlyapplied to a primitive in a single rendering pass.